Summary

Changes

Please provide a summary of what's being changed

This PR updates the Kafka Envelope schema and logic by making sure that the result is a list of values rather than a list that contains a list of values, i.e. the result went from this [['{"key":"value"}']] to this ['{"key":"value"}'].

The linked issue contains an explanation of why the result was the way it was and also why that is a bug, but essentially a Lambda function can subscribe to a single topic/partition and thus there's no need to accommodate multiple lists of values since all values will inevitably belong to the same topic/partition combination.

While working on the PR, I also removed from the envelope the forced JSON transform of the values which was causing the envelope to not work with values that were anything other than a JSON-stringified string.

For example, the following payload, which has aGVsbG8= (aka "hello world") as value would not have worked because the envelope was always trying to run JSON.parse() on the decoded value:

{
  "eventSource": "aws:kafka",
  "eventSourceArn": "arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4",
  "bootstrapServers": "b-2.demo-cluster-1.a1bcde.c1.kafka.us-east-1.amazonaws.com:9092,b-1.demo-cluster-1.a1bcde.c1.kafka.us-east-1.amazonaws.com:9092",
  "records": {
    "mytopic-0": [
      {
        "topic": "mytopic",
        "partition": 0,
        "offset": 15,
        "timestamp": 1545084650987,
        "timestampType": "CREATE_TIME",
        "key": "cmVjb3JkS2V5",
        "value": "eyJrZXkiOiJ2YWx1ZSJ9",
        "headers": [
          {
            "headerKey": [104, 101, 97, 100, 101, 114, 86, 97, 108, 117, 101]
          }
        ]
      }
    ]
  }
}

The new logic will stop at the base64 decoding and not make assumptions on the actual contents of the string. Customers can still use the JSONStringified we provide to do the parse and transform if they know it's of an appropriate type. For example, they can parse this event where value is "eyJrZXkiOiJ2YWx1ZSJ9" (aka "{\"key\":\"value\"}"):

{
  "eventSource": "aws:kafka",
  "eventSourceArn": "arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4",
  "bootstrapServers": "b-2.demo-cluster-1.a1bcde.c1.kafka.us-east-1.amazonaws.com:9092,b-1.demo-cluster-1.a1bcde.c1.kafka.us-east-1.amazonaws.com:9092",
  "records": {
    "mytopic-0": [
      {
        "topic": "mytopic",
        "partition": 0,
        "offset": 15,
        "timestamp": 1545084650987,
        "timestampType": "CREATE_TIME",
        "key": "cmVjb3JkS2V5",
        "value": "eyJrZXkiOiJ2YWx1ZSJ9",
        "headers": [
          {
            "headerKey": [104, 101, 97, 100, 101, 114, 86, 97, 108, 117, 101]
          }
        ]
      }
    ]
  }
}

like this: KafkaEnvelope.parse(event, JSONStringified(z.object({ key: z.string() }))) (which will work also when using the Middy.js middleware or decorator.

Please add the issue number below, if no issue is present the PR might get blocked and not be reviewed

Issue number: fixes #3488

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.